Recently, development of information processing apparatus has been significant, so that novel apparatus such as a visual telephone and a system for television conference is being proposed. In a field of image transmission such a in the television conference and visual telephones, a basic technique therefor is a technique for compression and decompression of moving picture. Presently, international standardization work is being carried out at MPEG1 (Motion Picture Expert Group part 1) and MPEG2 (Motion Picture Expert Group part 2) of ISO (International Standard Organization). MPEG1 standard is a transmittal standard for non-interlace image whose bit rate is comparatively low, whereas MPEG2 is the transmittal standard for non-interlace or interlace image whose bit rate is comparatively high.
Though there are proposed various techniques for compressing and decompressing the image, one among those is a motion compensation predicting encoding technique. First, an inter-frame prediction encoding technique will be explained. The inter-frame prediction encoding is the encoding in which time correlation of the image is utilized, and difference between previous image and present image is transmitted in a sending side, and a transmitted difference value is added to the previous image stored in a receiving side so as to reconstruct the present image. A process therefor is shown in equation (1). EQU D(x,y,t)=D(x,y,t-1)+.DELTA.D(x,y,t) (1)
In this method, a transmission bit width can be reduced for an image whose change or motion therefor is small, (in other words, the image in which correlation is high in terms of time direction). However, the bit width may be increased for the moving images. Thereby, a process of the motion compensation is added to the moving images, so that the bit width is reduced. In the motion compensation process, the moving quantity of the previous image and present image is evaluated in the sending side, and its quantity (motion vector) is transmitted together with the difference value. In the receiving side, the difference value is added to a value of position which is transited from the previous image by amount of the motion vector, so that a value for the present image is reconstructed. Its process is expressed in equation (2). EQU D(x,y,t)=D(x+mvx, y+mvy, t-1)+.DELTA.D(x,y,t) (2)
Here, mvx and mvy indicate the motion vector in x and y directions, respectively.
There are several methods for taking the motion vector. When accuracy of the motion vector is pixel unit, there exists pixel's value D (x+mvx, y+mvy, t-1) for position that the motion vector points (see FIG. 9(a)). However, when the accuracy of the motion vector is 1/2 unit, there may be a case where there does not exist a pixel in a position that the motion vector points (see FIG. 9(b)). In that case, that point will be obtained by interpolating neighboring 2 or 4 points (see FIG. 9(b)). An interpolation process by four points is indicated in equation (3). EQU D(x,y,t)={D([x+mvx], [y+mvy], t-1)+D([x+mvx]+1, [y+mvy], t-1)+D([x+mvx], [y+mvy]+1, t-1)+D([x+mvx]+1, [y+mvy]+1, t-1)}/4+.DELTA.D(x,y,t)(3)
Here, [x] indicates a maximum integer which does not exceed rational number x.
As described above, in the image decompressing apparatus by the motion compensation prediction encoding, necessary are the interpolation process of reference image and addition with the difference value (hereinafter referred to as reconstruction process).
As a structure for the image to be transmitted, there are available a frame structure and a field structure. Referring to FIG. 10(a), a scanning line is transmitted for each line. In the field structure, an interlace scanning is performed as shown in FIG. 10(b), so that even-number line is transmitted in a first field while odd-number line is transmitted in a second field.
Moreover, as a method for calculating the motion vector, there are a frame prediction method and a field prediction method. In the frame prediction, a reference image is treated as the frame structure, and in the field prediction the reference image is treated as the field structure (when the reference image is of the frame structure, the image is treated as of the even-number line or the odd-number line alone).
In MPEG1 standard, only the frame structure is adopted. An example of a decompression circuit in the course of motion compensation prediction encoding in that case is shown in FIG. 11. In FIG. 11, an adder 2 is a block where the reference image is interpolation-processed, and an adder 3 is another block where the reconstruction process is performed. A reference image memory is a frame memory. FIG. 12 shows a timing chart. For difference data transmitted at any time, a reference image corresponding thereto is accessed at ay time, and is interpolation-processed at the same time with data readout, and then a reconstruction process is performed simultaneously after the interpolation process. As shown in FIG. 12, these processes are performed continuously, so that the interpolation process and the reconstruction process are carried out same time. As for the difference data, they are inverse-quantization-performed at computing element 6, and processes such as an inverse cosine transformation are carried out, so that delay corresponding thereto is caused. Thereby, it is necessary to adjust timing for accessing the reference image, in accordance with the outputs of these processes' results.
For images transmitted by signals according to MPEG2 standard, both frame and field types can be adopted for image structure for transmitted difference and for prediction methods. In an image compress/decompress apparatus which handles signals where both frame and field types can be adopted for the image structure and the prediction methods as in MPEG2, there must be provided a frame/field converting unit prior to the reconstruction process so that the image structure and the prediction method for the difference are matched. An example of a decompression circuit for the motion compensation prediction encoding in that case is shown in FIG. 13. In FIG. 13, an adder 2 is a block to perform the interpolation process of the reference image, and an adder 3 is another block to perform the reconstruction process. Moreover, frame/field conversion is carried out in an intermediate memory. The reference image memory is a frame memory. A process content where an entire frame is divided into small image planes (for example, 8.times.8 block) will be explained utilizing a timing chart shown in FIG. 14. The small image planes and peripheral pixel thereof are read out from the reference image memory, and interpolated in the adder 2 and their result is written to the intermediate memory 4. The difference data are inverse-quantization-performed in the computing element 6 and processes such as the inverse cosine transformation are carried out. These processes are performed simultaneously during period (1) of FIG. 14. After these processes are completed about the small image planes, the reference image is read out from the intermediate memory 4 in the same frame/field structure with the difference data, so as to carry out the reconstruction process. These processes are simultaneously performed during period (2) shown in FIG. 14.
Accordingly, in the image decompression apparatus having no intermediate memory therein, where the reference image is read in at the time of the difference data process, so as to perform the interpolation process and the reconstruction process, it is required to have an adder for use with interpolation and another adder for use with image reconstruction, so that amount of hardware has been quite a bit. Moreover, in the motion compensation prediction encoding image decompression apparatus having the intermediate memory therein, where the frame/field conversion process is carried out, not only the amount of hardware is large, but also there is no overlapping during operation time period of adders provided around the intermediate memory, so that one adder is not used while other adder is in use, thus being a very low-efficient system.
On the other hand, recently, there has been developed apparatus which compresses/decompresses digital codes by a digital signal process. In particular, in order to transmit/store the digital images having great amount of codes, worldwide standards such as MPEG, JPEG and H261 have been set, and there have been developed many compress/decompress units in accordance with these standards. In these standards, as a code compressing technique there are utilized motion compensation which utilizes a variable-length code and correlation in the direction of time axis, and a discrete cosine transformation. As a decoder, accompanied by the use of variable-length code, there is necessitated a code input buffer which temporarily stores an input code that is compressed in order to take synchronism of image and compressed code. Furthermore, accompanied by the use of the motion compensation, it is necessary to store a frame image in the past or in the future as an reference image memory. Moreover, since the input code is provided as a block unit to perform a second dimensional DCT, regardless of output being interlace/non-interlace, it is required to have an frame memory in order to output the decoded image in interlace or non-interlace mode.
Namely, three types of memory units are necessary which consist of a code input buffer for taking synchronism, a reference image frame memory for use with the motion compensation and a display image frame memory for use with conversion to frame/field. Since these memories are difficult to be integrated in a single LSI having the decode unit thereon due to a current limit in integrity degree in the event that a decode unit is to be constituted by a single LSI, they will have to be provided as an external memory which locates outside the LSI. However, if different memory is allocated to each different usage, the number of wirings, the scale of circuit and the amount of memory will become unduly large, so that in many case a memory serving as the external memory is used allocatedly and commonly as the buffers for above-mentioned reference image, display image and code input.
Usually, in the event that the motion pictures are compressed-encoding-series-decoded there occur five types of memory accesses: to write an input code in a code input buffer, to read out of the code input buffer, to read out the reference image due to the motion compensation, write into the reference image that reconstructs the reference image after decoding, and to read out the display image that displays the decoded image. The buffer can not overflow or disappear in the course of writing the ceaselessly data-flowing input code into the code input buffer and reading out the display image in order to constantly display the image, and reading and writing of the reference image is difficult to stop once started, due to overhead of the processing.
In order to process these memory access requests for the single external memory, two methods are conventionally adopted. One is a priority order method where, whenever the necessity of memory access occurs, the access request is sent to a bus arbitration unit, so that the bus arbitration unit performs the arbitration on the requested signals generated at each block according to predetermined priority order. Moreover, another method is a polling method where each request is processed in turn and without the priority order. In the former case, there is no problem when there are less memory accesses against the bandwidth of the memory, however, frequency of waiting state is increased when amount of the memory accesses is large so that the the memory access having low priority order is not likely to be processed. Thereby, overhead of the processing becomes large and the decoding of one-frame image can not be completed within one-frame display time, so that there is caused a problem where the input code does not synchronize with the output image. In the latter case, since a consecutive memory access can not be carried out, the overhead for page changeover becomes enlarged in a case utilizing DRAM for the external memory, so that the bandwidth may come short of.
On the other hand, there are two redundancies for the moving image:
(1) spatial-axis direction - - - resembles to a neighboring pixel, human eyes can not capture fine things very clearly;
(2) time-axis direction - - - consecutive images are very similar to each other.
In order to compress these redundancies of (1) and (2), there are taken the following techniques:
(1) As a certain size of block is taken as a unit, it is converted to a frequency component by performing a discrete cosine transformation (DCT), so that a high frequency component thereof is reduced by performing quantization;
Difference (DPCM) from the neighboring block is regarded as data; (2) As a certain size of block is taken as a unit, the motion vector and error component alone are regarded as data, by the motion compensation. PA1 decoding means for decoding an input code and restoring an image; PA1 input code write means for receiving a code series input externally and writing the code series to an external memory; PA1 input code readout means for reading out input series stored in the external memory, and transferring the code series to said decoding means; PA1 image reconstruction means for reading out a reference image stored in the external memory, and adding to the image restored in said decoding means so as to form a display image; PA1 image write means for writing the image formed in said image reconstruction means to the external memory; PA1 display image readout means for reading out the image written by said image write means from the external memory, in order to display the image in an external portion; PA1 connecting means for connecting said image reconstruction means, said input code write means, said input code readout means and said display image readout means to the external memory; PA1 data storing means for temporarily storing data which each said input code write means, said input code readout means and said display image readout means read out from the external memory; and PA1 memory bus arbitration means for determining to select which among said image reconstruction means, said image write means, said input code write means, said input code readout means and said display image readout means is to be connected to said connecting means; PA1 wherein said memory bus arbitration means divides five types of memory accesses including writing the input code into a code input buffer, readout from the code input buffer, readout accompanied by the motion compensation, writing into a reference image which reconstructs the reference image after decoding and display image readout for displaying the decoded image, into a first group including writing the input code into the code input buffer, the readout from the code input buffer and the display image readout for displaying the decoded image which present a higher access priority order and a second group including the readout accompanied by the motion compensation and the writing into the reference image which reconstructs the reference image after decoding which present lower priority order and are to be executed continuously, and PA1 wherein a right to access is given only in a predetermined period by a polling method in the first group, and the right to access is moved to the second group when all requests in the first group are completed; when the right to access is moved to the second group and if there is an access request within the second group, the memory access is executed to a memory in question; once the right to access is given thereto, the memory access within the second group is such that other access request is turned down until a series of memory access is completed.
If this is applied to a general image, the motion compensation is difficult to be applied to a portion having relatively much redundance in the spatial-axis direction (i.e. fast moving image), therefore, the DCT plus quantization and the DPCM are performed in order to compress the redundancy in the spatial-axis direction. In this case, the DPCM is carried out on a DC component having highest correlation with the neighboring block.
On the other hand, the motion compensation is performed on a portion having relatively much redundancy in the time-axis direction (i.e. image not so moving fast), and the motion vector and error component alone are regarded as data. In order to further compress the redundancy in the spatial-axis direction, the DCT plus quantization are carried out also on the error component in a unit of a certain block. However, since it is considered that the error component does not present a high spatial correlation to the neighboring block, the DPCM is not performed.
In brief:
(1) Motion busy image
DC component of DCT coefficient - - - DPCM
AC component of DCT coefficient - - - DCT+Quantization
(2) Less moving image
Motion compensation+DCT+Quantization
FIG. 15 is a conventional image processing unit that reconstructs the thus-coded data to the image. First of all, a decoding circuit 30 decodes a variable-length code, so as to obtain coefficients of flag and DCT and the like. Next, in a case of a block on which the motion compensation is performed, an inverse quantization (IQ) and inverse discrete cosine transform (IDCT) circuit 31 performs the IDCT on the DCT coefficient of the difference data so as to obtain the difference data, and simultaneously, a motion vector reconstructing circuit 32 reconstructs the motion vector by using such an algorithm shown in FIG. 16. Thereafter, utilizing the motion vector, a previous, future or both image data are taken from an image memory 34 by a motion compensation circuit 33, and are added to the difference data in an addition circuit 35 so as to form reconstructing pixel data.
In a case of a block where the pixel data themselves are DCT-performed without performing the motion compensation, the DC component alone of the DCT coefficient is DPCM-processed by a DPCM processing circuit 36 shown in FIG. 17. Then, the IDCT is performed in the IQ.multidot.IDCT circuit 31 along with other DCT coefficients, so as to form the reconstructing pixel data. The above-mentioned shows the conventional method for reconstructing the encoded image data to the image data. The meanings of commands shown in FIG. 16 and FIG. 17 are explained in FIG. 18.
As described above, in the conventional image decompression apparatus having no intermediate memory therein where the reference image is read in at the time of the difference data processing so as to perform the interpolation process and the reconstruction process, the scale for the circuit is unduly enlarged. Moreover, in the image decompression apparatus having intermediate memory therein where the motion compensation prediction encoding image decompression is performed accompanied by the frame/field conversion, there are caused problems where the circuit scale for the adder is unduly enlarged and there exists time duration during which the adder is not used.
Moreover, in the above-described conventional general bus arbitration function, there is caused a problem where a huge overhead is caused in the event of accessing to the external memory.
Moreover, as described above, the reconstruction of the motion vector and the DPCM process of the DC component in the DC coefficient necessitate to have a certain sequencer in order to judge conditions and perform each calculation using decoded flags shown in FIG. 16 and FIG. 17, thus resulting in a large scale of hardware. Moreover, since in the compression and decompression technique of the image data where various standards and application are possible, since there is possibility where interpretation of the flag and manner of the calculation may differ, it is desirable that above-described sequencer can be controlled by software so as to present more flexible applicability and extendability. However, in that case, an instruction memory therefor and so on will be necessary, so that the amount of hardware will be increased. In the conventional image processing apparatus two sequencers have been required, so that the amount of hardware tends to be unduly increased.